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The NASA program and project managers of 
the 1990s will continue to work in the envi- 
ronment of constrained resources in terms of 
reduced budgets, limited staffing and tight 
schedules. In a speech to the Explorers Club 
in January 1989, former NASA Administra- 
tor James Fletcher stated: 

The funds being requested do not permit 
us the luxury of backups, of alternatives, of 
programmatic robustness. Virtually every 
element of the program is being pursued 
on a success schedule — and we know in 
advance that there will be unforeseen tech- 
nical problems to solve and dilemmas to 
face which will require internal adjust- 
ments and constraints. 

In this environment there are focused efforts 
to improve program and project manage- 
ment. One potentially powerful tool avail- 
able to the project manager which has been 
used successfully in many government agen- 
cies is performance measurement. 

Performance measurement is a management 
tool for planning, monitoring and controlling 
all aspects of program and project 
management — cost, schedule and technical 
requirements. It is a means (concept and ap- 
proach) to a desired end (effective program 
planning and control). To reach the desired 
end, however, performance measurement 
must be applied and used appropriately, with 
full knowledge and recognition of its power 
and of its limitations — what it can and can- 
not do for the project manager. 

Performance measurement is not a new con- 
cept to the government or to the aerospace 
industry. It has its origins in the Department 
of Defense (DoD) programs of the 1960s. In- 
terest and application of the performance 
measurement concept spread to other gov- 


ernment agencies in the 1970s and 1980s. 
Today performance measurement is being 
applied to major programs of the DoD, Na- 
tional Security Agency, Department of Ener- 
gy, Federal Aviation Administration and 
NASA. Performance measurement is widely 
endorsed as a valid approach to controlling 
contract performance. 

The Goddard Space Flight Center (GSFC) 
has been implementing performance mea- 
surement system (PMS) requirements since 
1983 on major research and development 
(R&D) contracts with a price of $25 million 
or more and a period of performance longer 
than one year. GSFC’s PMS policy was estab- 
lished by the Center Director to provide for 
consistent application on all major Center ac- 
quisitions. Use of performance measurement 
is also encouraged on R&D contracts in the 
$10-25 million range, but applied on a case- 
by-case basis. GSFC currently has 12 con- 
tracts in various project phases that have 
PMS requirements. With the large number 
of major independent spacecraft and instru- 
ment development contracts at GSFC, such 
as the various meteorological spacecraft and 
instruments of the Geostationary Operation- 
al Environmental Satellite and Television 
and Infrared Observational Satellite pro- 
grams, we have had the opportunity to con- 
tinually improve our implementation of PMS 
through a “lessons learned” approach. Some 
of the more effective PMS applications have 
been on the Gamma Ray Observatory and 
the Tracking and Data Relay Satellite Sys- 
tem spacecraft contracts. 

What is the potential of this management 
tool? What does performance measurement 
do that a traditional plan vs. actual tech- 
nique cannot do? Performance measurement 
provides an improvement over the customary 
comparison of how much money was spent 
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(actual cost) vs. how much was planned to be 
spent based on a schedule of activities (work 
planned). This commonly used plan vs. actu- 
al comparison does not allow one to know 
from the numerical data if the actual cost in- 
curred was for work intended to be done. 

With performance measurement, actual 
work progress (work done, also known as 
earned value) is quantified by an objective 
measure of how much work has been accom- 
plished on the program. This added dimen- 
sion of a quantitative assessment of work ac- 
complished allows for comparisons to be 
made between the value of work that was 
done vs. the work that was planned to be 
done (schedule variance). It also allows for a 
comparison of the actual cost of work that 
was done vs. the planned value of the work 
that was done (cost variance). This analysis 
then provides for early identification and 
quantification of cost and schedule problems. 
A graphic depiction of the data available 
from the traditional plan vs. actual tech- 


nique compared to those available from a 
performance measurement system may serve 
to more clearly illustrate the concept. A hy- 
pothetical spacecraft program is expected to 
take five years to build at a cost of $500 mil- 
lion. Figure 1 shows the traditional plan vs. 
actual technique. If “time now” is the com- 
pletion of year 2, the graph indicates that we 
had planned to spend $250 million. The actu- 
al cost (i.e., time card charges, material ex- 
penses, etc.) reported to the government is 
$200 million. 

What can a project manager conclude from 
this information? Is it possible to determine 
if this program is overrunning or underrun- 
ning? With this limited information avail- 
able, a project manager may assume that the 
contract is underrunning and would have no 
basis to question the assumption that this 
program will underrun at completion. At a 
minimum it currently appears that the $500 
million funding estimate is adquate to com- 
plete this effort. 


• IN 

MILLIONS TIME 



148 


Figure 1. Traditional Plan vs. Actual Technique 
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In Figure 2 an additional data point has been 
added to the same hypothetical spacecraft 
program. The contractor has assessed the 
value of the work accomplished (or earned 
value) to date. This new information reveals 
that of the $250 million of work planned to be 
done to date, only $150 million has been 
done. Some work that was planned to be done 
has not been done and is reflected as a $100 
million schedule variance. Also, the $150 
million worth of work done can be compared 
with the actual cost of $200 million. 

This comparison shows the planned value of 
the work vs. the actual cost of that same 
piece of work. Now the project manager can 
see that this program is actually overrun- 
ning by $50 million to date. We now have 
enough data to question the validity of the 
$500 million funding estimate for completion 
of this effort. We can begin to see that this 
program is headed for an overrun of costs at 


completion along with potential schedule 
slippage. 

As a result, the project manager having the 
PMS data available in Figure 2 is better able 
to estimate early the total costs and projected 
period of performance of this program, there- 
fore avoiding a surprise overrun much later 
in the program. If the data yield a “doom and 
gloom” assessment, there is opportunity to 
make decisions early to avoid an approach 
that is too costly or that takes too long. The 
basic objective of performance measurement 
systems is to provide a suitable basis for re- 
sponsible decision-making by both the con- 
tractor and the government management by 
ensuring that (1) the contractor is using ef- 
fective internal cost and schedule manage- 
ment control systems, and (2) the govern- 
ment can rely on valid, timely and auditable 
data to be produced by those systems to de- 
termine program status. 
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Figure 2. Performance Measurement Technique 
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Unfortunately, there has not been a consis- 
tent experience within the Agency regarding 
PMS implementation. Personnel at various 
NASA Centers and in the aerospace industry 
believe that while some NASA applications 
of PMS have been successful and effective, 
other attempts to use PMS as a management 
tool have actually been counterproductive. In 
some instances, performance measurement 
systems have not always provided accurate 
reporting of cost and schedule status, and 
there are differing opinions about why PMS 
did not work in these instances. The most 
prevalent of these is that in the NASA envi- 
ronment and culture, a disciplined approach 
to program management is not appropriate 
or applicable. 

While it is healthy to question the worth and 
applicability of PMS for NASA programs, it 
is also beneficial to explore some of the com- 
mon sense features of PMS that have proven 
effective in controlling project costs and 
schedules in many government agencies for 
the past 22 years. 

Some Basic Principles 

Performance measurement can work for you 

if you apply some basic principles. 

• Plan the entire contractual effort. It is es- 
sential to plan the work for the entire pe- 
riod of performance. Near-term work is 
planned in detail while future work can 
be planned at a summary level. Failure to 
recognize all of the work to be done makes 
it impossible to properly allocate re- 
sources. Programs could consume too 
many of the resources on the near-term 
work and not leave enough to do the work 
downstream. 

• Maintain baseline integrity. The measure- 
ment of actual conditions against a disci- 
plined or controlled plan reveals perfor- 
mance trends that can help to predict fu- 
ture conditions and to determine a future 
course of action. 


• Determine accomplishment at the level at 
which the work is performed. Who can 
better assess the work that has been done 
and the work remaining to be done than 
the manager responsible for performing 
the work? 

• Measure accomplishment objectively. The 
most valuable status assessment of a 
piece of work is based on pre-defined 
milestones as opposed to personal feelings 
and prejudices lacking reality or sub- 
stance. 

• Summarize for higher levels of manage- 
ment. While accomplishment is assessed 
at a relatively low level, summary report- 
ing to higher levels of management, 
where resources are made available, is 
also essential for control. 

• Analyze variances and forecast impact. 
Variances are simply indications that ac- 
tual conditions are different from the 
original assumptions, and variances may 
indicate the existence of current or poten- 
tial problems. Analysis of the variances 
allows management to correct problems 
or to redirect efforts to avoid potential 
problems, as well as to project cost at com- 
pletion. 

In summary, the concept of performance 
measurement is good, common sense pro- 
gram management that NASA project man- 
agers have always practiced, but perhaps not 
in a formal way. 

Specifying Customer Requirements 
NASA authority for performance measure- 
ment is based on the agency requirement 
specified in NASA Management Instruction 
9501.1 “NASA Contractor Financial Man- 
agement Reporting System” and NASA 
Handbook 9501.2B Procedures for Contractor 
Reporting of Correlated Cost and Perfor- 
mance Data. The NASA Form 533P (where 
“P” represents performance) has been used 
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by contractors to report performance data to 
NASA, unless the contractor has another for- 
mat that serves as the equivalent. The 533P 
is essentially a minimum NASA require- 
ment for data reporting purposes only. It 
does not require that an identifiable system 
or set of subsystems support the data. As the 
contractors are free to generate data in any 
way they desire, there is the high potential 
for invalid or misleading data if this is the 
only requirement placed on a contractor re- 
lated to performance measurement. Without 
a system requirement for visibility and con- 
trol of the baseline, for objectivity in measur- 
ing accomplishment, or for discipline in fore- 
casting estimates to completion, performance 
measurement may not yield valuable infor- 
mation. While data can be reported on a 
533P, a more disciplined approach to the 
management system is needed to identify 
some rules for performance measurement 
systems. These rules are known within the 
government and aerospace industry as the 
“criteria.” 

The performance measurement criteria do 
not identify a specific management control 
system to be applied to a program; rather, 
they represent a set of standards against 
which to measure the acceptability of a con- 
tractor’s cost and schedule control system. 
There is, in fact, a variety of equally effective 
ways for contractors to meet the criteria re- 
quirements. The criteria allow a company to 
organize in any way that suits the company’s 
philosophy and style. The criteria also allow 
a company to develop any desired policies, 
procedures or methods that meet the require- 
ments. The criteria address the age-old ques- 
tions of any project manager: What work is to 
be done? Who will do it? When is it going to 
be done? How much will it cost? Where is the 
program heading? What has changed? 

The contractors address these questions 
through their management systems’ inte- 
grated set of subsystems. These are subsys- 
tems that would be required to manage a pro- 


gram whether or not a performance measure- 
ment requirement was imposed. Perfor- 
mance measurement criteria simply require 
that a more disciplined approach be applied 
to each subsystem. The PMS subsystems are 
(1) work authorization, (2) budgeting, (3) 
scheduling, (4) data accumulation, (5) vari- 
ance analysis and estimate at completion, (6) 
subcontract and material control and ac- 
countability, (7) indirect expense manage- 
ment, and (8) change baseline control. PMS, 
then, does not address just the accounting 
system, but rather it addresses the integrat- 
ed set of subsystems that constitute all ele- 
ments of program planning and control. 

A Good Management System 
The key to the power of performance mea- 
surement is that performance measurement 
data are only as valid as the management 
system that provides them. If a contractor op- 
erates a sound internal management system, 
the customer should be able to extract sum- 
mary data from that system that reflect pro- 
ject status. To have a valid management sys- 
tem applied to NASA work in contractor 
plants, several conditions need to be met. 

First, a management commitment from the 
top down is required — all levels of manage- 
ment support are essential. It is not enough 
to have project financial or resources support 
personnel discussing PMS with the contrac- 
tor. The involvement of technical personnel 
is critical. PMS involves all aspects of pro- 
gram management and needs to be viewed in 
this way by NASA project and functional 
management personnel to be effective. 

Second, management system discipline must 
be stressed and required. While it may be de- 
sirable to maintain a spirit of cooperation 
and non-adversarial relations with our con- 
tractors, PMS is not of any value without a 
disciplined approach to management. With- 
out a requirement for the contractor to main- 
tain a baseline, to apply objective techniques 
for performance measurement, or to reliably 
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forecast the cost to completion, there can be 
no confidence in the value of the data that 
the management system generates and that 
the contractor reports to NASA on a monthly 
basis. 

Third, use of data generated by the PMS is es- 
sential. A few simple mathematical formulas 
and computations yield very revealing infor- 
mation about the project status and potential 
future of the program. Use of data serves to 
facilitate communications internally and be- 
tween NASA and the contractor. 

Fourth, corrective action needs to be taken 
when problems are identified. A manage- 
ment system supplies data points, not solu- 
tions. It provides visibility into cost, schedule 
and technical status. A system, however, 
does not manage the project, people do. A sys- 
tem cannot eliminate schedule slippages or 
stop overruns, but it can help the project 
manager to understand the potential impact 
if trends are allowed to continue without 
mid-course correction. 

Fifth, an in-plant review of the contractor’s 
management system applied to a program 
and conducted by a NASA team of interested 
and knowledgeable technical and resources 
personnel is critical. The NASA personnel 
gain invaluable knowledge of the policies, 
methods and procedures used by the contrac- 
tor to generate monthly status reports. By 
understanding the source of the data, we can 
calibrate the validity of our monthly custom- 
er reports and require the contractor to re- 
vise procedures that do not produce valid 
data. 

PMS is not intended to replace traditional 
m a n agement tools — it should enhance them. 
Day-to-day program management is essen- 
tial. In fact, if managers are relying solely on 
performance measurement data generated at 
month-end, they will be learning of problem 
situations much too late to be effective. Peri- 
odic status reviews, “kicking the tires,” and 


routine communication internal to the con- 
tractor and between the contractor and gov- 
ernment managers are critical in managing 
a program. PMS may identify a new problem, 
but in most cases, it allows quantification of 
a known problem through all elements of the 
work breakdown structure and through the 
functional organizations to provide a basis 
for improved management decisions. 

Cost Effectiveness 

In times of constrained resources it is reason- 
able for managers to question the cost effec- 
tiveness of PMS. What are the benefits and 
associated costs? The question is difficult to 
answer, however, since both the benefits and 
costs are nearly impossible to quantify. 

PMS results in a better controlled project 
with improved communication, both inter- 
nally and with the customer. To quantify the 
benefits is to ask, “What is the value of good 
management?” It is not evident how a cost 
savings (or cost avoidance), a shortened 
schedule, or improved technical performance 
through corrective action can be clearly asso- 
ciated with results or a specific cost. 

The costs of PMS have also defied quantifica- 
tion for 22 years. The PMS-unique costs on 
the total contract cannot be separately iden- 
tified from the management costs that would 
be incurred in any case. They are not rou- 
tinely collected by contractors, nor is it con- 
sidered practical to do so. This was illustrat- 
ed in a 1987 survey of GSFC contractors who 
had implemented a PMS requirement. In the 
survey, some contractors suggested that the 
costs of PMS beyond the usual management 
costs may be expressed as a percentage rang- 
ing from 2 percent to 6 percent of total con- 
tract costs. In each case, however, the con- 
tractor could not substantiate the percent- 
age. It was someone’s “non-scientific esti- 
mate,” as stated by one contractor. Surveys 
conducted by the DoD show that there is no 
correlation between the cost of PMS and the 
contract costs. 
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This is not to say that there cannot be costs 
associated with PMS requirements. In fact, 
the cost of implementing PMS is in direct 
proportion to the quality of the existing man- 
agement system. The poorer the state of the 
contractor’s system, the greater the need for 
improvement and the more it will cost to im- 
prove. Contractors who maintain discipline 
in their systems would incur very low costs to 
implement PMS on subsequent contracts. If 
the same contractors did not maintain their 
systems, over time the cost to implement 
PMS on future contracts would be greater as 
the need for improvement became greater. 

Further, if there is not an existing integrated 
cost and schedule management system, the 
contractor will certainly incur costs to devel- 
op one. GSFC experience, however, has been 
that contractors awarded major development 
procurements that contain PMS require- 
ments are contractors who already have 
operational PMS systems as a result of their 
dealings with DoD. Costs of PMS have been- 
minimal compared to the significantly great- 
er value added. 


There is one additional factor to consider in a 
discussion of the costs of PMS. Typical points 
of contention between the government and 
industry concerning PMS implementation 
include the levels of detail identified for 
management and reporting, and the vari- 
ance analysis thresholds identified for cus- 
tomer reporting. It is possible to avoid incur- 
ring unnecessary cost to the government and 
frustration for the contractor by not request- 
ing reports that no one reads or uses, or “nice 
to have” items or analyses. 

In summary, with the focus on efforts to im- 
prove program and project management, 
PMS is a potentially valuable tool. Like any 
tool, however, it is only as valuable as the 
user chooses to make it. Implemented proper- 
ly, PMS can ensure the generation of valid 
cost and schedule performance data to ease 
the manager’s decision-making process, 
which can result in more effective program 
planning and control. 
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